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Energy/Environmental Programs 
and Strategic Priorities 

Energy/environmental programs are critical to 
meeting governments' strategic priorities 

• U.S. Dept. of Energy (DOE) Hydrogen Program 

- Climate change (reduce greenhouse gas emissions) 

- Energy security (reduce dependence on oil) 

• Energy efficiency programs 

- Climate change ("low-hanging fruit" solution) 

- Energy security (reduce oil imports) 

- Economic competitiveness (save money) 
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Evaluations Help Programs Achieve 

Intended Outcomes ^ 

• Increased emphasis on evaluation by White House Office 
of Management and Budget (0MB) 

- Achieve intended results efficiently and effectively 

- Strengthen the design and operation of programs 

• Evaluations help maximize programs' opportunities to 
achieve strategic goals 

- Inefficient/unproductive programs represent "lost 
opportunities" to achieve chmate and energy goals 

- Evaluations inform decisions that increase program 
productivity, enhance fiscal responsibility, etc., which improve 
chances of achieving intended results 
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What Decisions are Informed by 
Program Evaluations? 

Evaluations provide information to inform decisions 
made during the critical program design, redesign, 
budget and planning cycles 

Decisions (examples from case studies): 

- Whether to continue a project as is or with adjustments made to further 
improve the hkehhood of success 

- Which processes need reform/streamhning and how to do so 

- Whether to terminate an entire program area or individual projects 

- How much money to allocate to research areas and specific projects 

- Whether to redirect research of new priority areas or adjust research 
portfolio 

- Which sectors or segments of the population should be targeted 
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Project Background ^ 

• Raise the profile of program evaluation 

• Show how evaluation can help improve efficiency or 
achieve other program goals 

• Concrete examples in easy-to-read format 

• Invitations sent to hundreds of evaluators to submit 
potential case studies 

• Focus on energy and environmental programs 

• Information provided via evaluation reports, 
performance data and interviews 

• Projects were selected based on three criteria 
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Criteria for Case Studies 

Our goal was to find case studies that 
fulfilled the following criteria: 

1. Evaluation made specific observations or 
recommendations. 

2. Program took some action to implement 
recommendations. 

3. Benefits documented in follow-up evaluation or 
other report. 

Selected four cases plus included one previous 
case study developed by DOE 
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Case Studies 




Preview of Case Study Results and Decisions 
Informed by Program Evaluations 

• A U.S. DOE R&D program saved $27 million dollars by 
conducting annual peer reviews and identifying poor performing 
projects, redirecting funds to more productive R&D 

• A Wisconsin energy efficiency program increased CFL sales 
among women after evaluation found a significant gender gap in 
CFL sales 

• A U.S. EPA pesticide program will finish a large project four 
years sooner than previously estimated by implementing 
evaluators' recommended methods to increase efficiency 

• A U.S. EPA program saved 90% of a web communication budget 
by simplifying a web tool after an evaluation found it too 
complex for most users' purposes 

• A Quebec energy efficiency program enrolled new customers 
more quickly after evaluators suggested ways to streamline 
processes 



U.S. DOE Hydrogen Program Peer Reviews 



1. The U.S. DOE Hydrogen Program conducts annual peer 
reviews, using external independent expert panels, of 
hundreds of its projects while they are in-progress 

• Case study looked at 695 reviews of projects conducted over 
five years. 

2. After completion of the expert reviews, the following 
decisions were made: 

- 82% of projects were continued (received an average reviewer rating of 
3.0 on a scale 1 to 5) 

- 13% of projects were completed and not renewed 

- 5% of projects were discontinued (on average, a 2.7 rating) 

- Many (67%) of the low performing projects were continued after 
recommended modifications and adjustments were made to increase the 
likelihood of their success 



Expert-Informed Adjustments 
and Realignments Helped 
Improve the Likelihood of Success 

Peer-reviewed projects improved after adjustments were 
made fc)llowing the initial peer reviews. «^ 

— [based on analysis pairwise comparison of mean value ratings from 
two separate reviews] 

Of the projects that were continued despite an initial low 
rating, 31 were reviewed at least once more in this case 
study's timeframe. 

— Of these, 81% had a higher rating at their next review. 

— On average, the 31 projects rated 2.6 at their first review, and 3.0 at 
the next review, (statistically significant) 



Information from the Peer Reviews Helped 
the Program Save Millions of Dollars 



4. Sizable financial savings was achieved by discontinuing 
poor performing projects deemed unlikely to contribute 
to successful mission accomplishment 




At a total cost of $1.8 million 
over the years 2003-2007 to 
conduct 5 annual reviews of 
projects, the Program 'saved' 
nearly $30 million 

- ["saved" = avoided continued 
investments that are redirected to 
higher valued projects] 

- a fifteen-fold direct return on 
investment in peer reviews 

- Net savings of $27 million 



V) 

C 

o 



n 
"o 

a 



Figure 1. Cumulative Peer Review 
Cost and Benefits: DOE Hydrogen 
Program, 2003-2007 
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WI Focus on Energy CFL Program 

1. Evaluation in 2003 found large gap among males 
and females participating in CFL rebate program 

• Concurrent research showed 
women more hkely to 
purchase hght bulbs in 
grocery stores, drug stores, 
general retail, etc. 

• Most of program's 
participating stores were 
home improvement/hardware 
stores, where men tend to 
shop 

2. Based on these findings, 
program adjusted tactics: 

• Added more retail partners 

• Targeted advertising 



Figure 1. Gender Participation, FY2003 
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CFL Sales Increased Among Women 
After Program Adjusted Tactics 

3. Later evaluations found statistically significant 
increase in CFL sales among women compared to 
2003 » 



Gender gap shrank for 
rebate program while 
overall sales increased 



Sales increased at new 
retail partners by 68% 
from 2006 to 2007 



Figure 3. CFLs Sold Through Program 
Among New Retail Partners 
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U.S. EPA Pesticide 

Product Re-registraton 

U.S. EPA required to review all pesticide active 
ingredients registered before November 1, 1984 

1. Process evaluation in 2005 found that the U.S. EPA 
Office of Pesticide Programs (OPP) was on pace to 
finish product re-registrations in 2018 

• Five years longer than was budgeted 

• Evaluators made 21 recommendations to streamline the re- 
registration process 

2. Managers implemented 17 of 21 recommendations 
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Evaluation Spurs Quicker Pace, 
Saving Time and Valuable Resources 



3. Efficiency jumped after implementation of 
recommendations ^ 

• Number of actions nearly doubled in less than a year J 

• OPP now on track to finish re-registrations by 2014 

- Four years sooner than estimated pace before program 
implemented reconmiendations 

• Allows OPP to move staff to other projects sooner 

• New labels applied to products faster 

- Updated health and safety information 

- Helps those applying pesticides in the field 
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Energy Star: Home Energy Advisor 



U.S. EPA spent $100,000 a year on online home energy 
auditing tool 

• Collects home information, reconmiends products to buy 
to increase energy efficiency 

1. Study found most participants would not use all 
of the web tool's sophisticated features 

• Web tool offered a "good starting point," but most would 
get other recommendations before buying products 

• Evaluators recommended that web tool offer "options to 
explore" rather than detailed recommendations 
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Substantial Savings Achieved by 
Simplifying Web Tool 

2. Evaluation prompted managers to 
eliminate web tool, linked web visitors to 
similar one by U.S. DOE ^ 

• U.S. EPA later developed a simplified tool 

3. U.S. EPA's new web tool cost only $10,000 
a year to maintain 

• 90 percent reduction in costs 
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Hydro-Quebec^s Empower Programs 

Empower Programs provide financial incentives _ 
to businesses for efficiency projects 

1. Evaluation recommended ways to streamline 
application process, among other concerns 

• Customer complaints about slow application process 

2. Program managers created task force to 
implement evaluators' recommendations 

• Program staff implemented steps to improve workflow 

- Cut amount of application documents in half 
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Adjustments H( 
Energy Saving 



3. More customers added 
at a quicker pace 

- Application times 
reduced 

- Fast track procedure 
for standard projects 

- Programs on track to 
meet energy savings 
targets 

- Customer satisfaction 
also improved 



Programs Achieve 
>als More Quickly 



Figure 1. Average Time Between 
Receipt of Application and Payment 
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Common Themes 




How Implementers in Case Studies 
Found Evaluations Useful 

• Detecting and quantifying problems 

- "You can't attack the problem until you've quantified it" 

• Resulting in actionable decisions 

- Prioritizing budget decisions 

- Eliminate or reform inefficient/unproductive elements 

- Enabling goal achievement (Higher customer satisfaction; 
Reaching more participants; Services better targeted; Reach 
program goals sooner 

• Motivating staff to improve 

- Demonstrates management is serious about c^^stiflg, 
problems W ^^^'"SV 'nstitute 
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Some Keys to Success 



Communication between evaluators and 
implementers 

- Each side feels they are a part of the entire process 

Following up 

- Subsequent evaluations or performance measures 

- Lacking in many evaluation efforts, but important for 
demonstrating success 

Making evaluations ''useful'' 

- Providing information/recommendations that 
implementers can act upon 



"Closing the Loop^^ 

— Evaluating, Implementing Recommendations, Following Up - 




Behavior, Energy and Climate Change Conference 
Sacramento, CA November 9, 2007 



Concluding Thoughts 




Evaluations are important for energy programs' success 

- Help achieve intended results efficiently and effectively 

Both e valuators and implementers have incentive to show 
success of decisions informed by evaluations 

- A matter of following up 

Communication and program redesign are continuous processes 

- Not a one-time lecture, but an ongoing conversation 

"Lost opportunities" to achieve climate and energy goals via 
policies and programs are avoided when we close the loop by 

evaluating, implementing recommendations, fqUowing up. 
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